The New Now Thing

Two years ago, there was no public market for Linux companies. The combined market cap of the entire category was zip. Then, in late '99, Red Hat, Cobalt, Andover and VA Linux ran up IPOs that ranged from impressive to stratospheric — all before Christmas. Then, after the turn of the Millennium, the market for Linux stocks fell like a bad tent. But not completely. Over the past few months (we're writing this on 9/12), the market floats along at apparently sane levels, where plain old volatility looks positively stable.

So, after one year, the total value of the Linux marketplace (for companies, if not for products) is in the many billions of dollars. And the world's appetite for Linux continues to grow steadily.

During the same period, countless new Linux companies and products came into the world, and large blue-chip technology players like IBM and HP declared their affection for Linux and its open source development model. Oceans of ink and galaxies of pixels were spent telling the story of Linux vs. Microsoft — penguins against the empire. Linus was cast as the latest David (replacing Marc Andreessen) doing battle against the same old Goliath.

Early this year Linus' employer, Transmeta, finally broke its silence and announced a new pair of chips — novel new microprocessors called Crusoe — built to run in low-power mobile computers and other specialized devices. Few mentioned that for years Linus had made no secret of his belief that Linux has a promising future in devices that looked less and less like PCs.

After the Big Announcement, there was a gradual fade in the buzz, and coverage went back to the usual preoccupations with workstations and servers. Could Linux ever win the desktop, which Microsoft owned since the Reagan administration? Would Linux' success as an OS for Web servers leverage into the traditional IT space, where Microsoft wasn't as strong and Sun was a bigger enemy?

Meanwhile nobody seemed to notice was that Linux became the leading operating system for embedded devices and applications. There was no battle. No CEO surrendered his keyboard to an army of penguins. There were no parades. And there was almost no press coverage whatsoever.

But it happened. Linux won the embedded space. The only other OS vendor of any significance in that space, Wind River, still makes a fine living doing what it has always done, which is supply semiconductor companies and various manufacturers with operating systems, software development tools and support, to the tune of a few hundred million dollars in revenue per year. Not bad. Also not significant.

Go back a few years and two other leaders in the space were Lynx and Ready Systems. Lynx is now Lynuxworks, and Jim Ready — perhaps the leading personality in the history of embedded operating systems — runs MontaVista, one of the top new Linux embedded OS companies.

Meanwhile Bryan Sparks, who made Caldera the first well-branded Linux distribution back in the mid-Nineties, is charging forward with Lineo, a Caldera spin-off that is on an aggressive campaign to blow away not only its Linux competition, but Wind River as well.

It's hard to find any hard numbers about Linux adoption in the embedded world, just as it was hard to find hard numbers about the Internet when nobody seemed to notice that it was taking over. But it's happening. Just as surely, and just as rapidly.

The embedded processing market is enormous. It's everything that processes bits and isn't general purpose computing. Look at it another way: it's everything that could possibly connect to the Net,: the sprinker system in your lawn, the conveyor belt controller at a distribution facility, the navigation system in an aircraft ... anything and everything that relies on digital intelligence. This is the world laying at Linux's feet. The whole damn place is Antactica now.

Why? The short answer is engineers. Hey, they like it. It's ideal for net-connected processing. It's familiar. It's small. It's net-native. It's fast. It's reliable. And it's open to improvement by anybody who wants to bang on some code. This population is rapidly growing to include pretty much every programmer who isn't working exclusively on Microsoft-based applications — group that can only get larger.

The embedded Linux company with the highest profile is Utah-based Lineo, which filed earlier this year to make an initial public offering (IPO), and is currently in its "quiet period." It's S-1 filing shows revenues in the few-million range, but growing at a very rapid rate. A $37 million investment by a group of Asian investors can't hurt. Money is also starting to pour in to others in the category, including $20 million to LynuxWorks and $9 million to MontaVista.

But the leading personalities are Lineo's Bryan Sparks and MontaVista's Jim Ready. While Sparks and his company have attracted the most ink and clearly have huge momentum, Ready and his company are rapidly building a strong reputation inside the Linux community for both its veteran sensibilities and its pure-play membership in the Linux and open source movements. While Lineo also has roots deep in the same movement, and continues to offer a growing assortment of open source products, it uses Linux as the most significant base component in a suite of tools and other products that also include closed and proprietary code. The two companies could hardly be more different and still competing in the same space for the same customers.

So we thought it would be a good idea to interview the two gentlemen, and get their unique perspectives on this Big Bang that is still just a few seconds old. The interviews took place on September 7 and 8, and ran a total of nearly 16,000 words — all of them interesting. We hope our edits retain the core points and insights both of them make about the marketplace they lead.

— Doc Searls

Jim Ready, MontaVista Software

How did you get started in embedded linux?

My first job back in '76 was with an early mil-spec computer company: Rolm.

This is the same Rolm that was later acquired by IBM

Right. That was my first job out of Berkeley. Back then it was all about cross-development, because the Rolm machines would go in airplanes, so you wouldn't be doing development on them. They would have development stations just like we've had for embedded work ever since. So many of the fundamental principles are the same. What changed were the form factors. These Rolm boxes were huge. Microprocessers have since blown all of that. But the fundamental lessons we learned a long time ago

So that was your first real job

Then we were fortunate in '81. We were working for this real good company, Rolm, But if you remember around 1980 was when the 68k was announced. It became clear that the minicomputer situation was going to change. You would be able to build good computers out of micropricessors. It took a while. Businesses take time to change. But fundamentally it became clear that the wave of the future was obviously the microprocessor. It would challenge the PDP-11s and soforth.

But you saw the embedded world changing.

Yes. Just as Rolm machines got embedded inairplanes and ships and everything else - factory automation and so on - the same thing would happen with computers based on microprocessors. So that all the system software that we had been used to on minicomputers - operating systems and tools and that sort of stuff - would need to be put in place for microprocessors. But there really wasn's a standard, off-the-shelf, constant-across microprocessors operating system. Everybody had proprietary unique-to-each-microprocessor architecture systems.

Each was essentially a stovepipe that went up from the silicon all the way up through applications.

If it existed at all. So we thought, why not build an RTOS - a real time operating system - that would, independent of processor architectures, present the same interface to the customer whether they were on an 8086 or a 68k. Then let them program in the languages that were popular at the time - PLM or PASCAL, since C came later - and let customers pick and choose without changing their devlopment environment. And that was the original Hunter & Ready solution, which is still valid to this day.

I remember Hunter & Ready, though I hadn't heard the name in a long time.

That was the original company that later became Ready Systems.

I guess I just shortened it myself to just Ready in my subconcious history of the category,

You know, there is this totally weird coincidence - except it probably isn't - that Colin Hunter became one of the founders of Transmeta, where Linus now does his work.

That is amazing. Are you and Colin still buddies?

Oh sure, yeah. We have the old friendship back. Colin has been extremely gracious to us, with this totally weird twist, where we run a Linux company and Linus works for Colin.

Is Colin one of the reasons Transmta hired Linus?

He is. I don't think there are any secrets or whatever, but as a quick aside, Colin was starting up Transmeta, needed some good system programmers, and a Swedish guy he had hired said he knew this guy in Finland. So Linus came over and got this day job working for this very secretive company. Originally Transmeta was not a Linux company, but eventually Linux did become important there. But Linus truly was hired as a great programmer for internal things, with no notion about what Linux would become today.

I didn' know that. I thought they had hired Linus at least in part for his Linux expertise.

They hired a fabulous programmer. But his work was related to the code-morphing software and other things. I don't think I'm talking out of school when I say that Colin said that Linus was the most brilliant guy he had ever met, and was a great programmer.

So you and Colin are still working together, in a way.

It's a very small world. In fact I knew, when I went off on the Montavista tangent, that our paths would cross again because they had to. It was just gonna happen. The ultimate irony is that Linus was involved too. By the way, everything we do is completely arms-length. We know people, and that's fine. It's all business.

But when people know people, things tend to happen. If Linus hadn't worked at Transmeta, chances are they wouldn't be doing a Linux-based chip.

Yeah. It's just an odd sidebar that our paths would intersect again perfectly around Linux.

It was an interesting fact to me that when I first met and interviewed Linus that he brought the conversation around to embedded. He did the same in his speeches. Wtthout giving away too much about Transmeta, he kept saying that the most interesting future to Linux wasn't in beige boxes. It was in little things in the embedded space.

Obviously he's a smart guy who can figure this stuff out, and maybe it was some influence from Colin. It doesn't matter. We certainly hold to that belief. The most level playing field, and the one that favors Linux the most, is the embedded space. If you go from the tyranny of Microsoft on the desktop to the fairly competitive market in servers - with Solaris and NT, where Linux also does well - and then to embedded, you see Wind River is the biggest player but still a tiny fraction of the overall market. That market is begging for standardization, for commonality and all the attributes that Linux provides. It is no surprise that if you look at conditions in the embedded market and its fragmentation, and the obscurity of even the most popular system which is Wind River's VXWorks, which is obscure relative to Linux, the field is just wide open. VXWorks is fine, but it's known only to its developers, and it's closed source and nobody knows how it works. So, as much as Wind River would like it to be mainstream, it isn't, nor is it going to be. So there is this pent-up demand in the embedded area to get normal. Linux is the shining light in this dark space. The customers are like moths to a flame, in a good sense. Linux is extremely desirable, if it will meet their technical requirements. Remember that these guys are engineers.

So you're saying this has been a norm-less space where absolute heterogeneity kept everything apart- until now.

Oh yeah. So everything interesting had to be imported from somewhere else, from the mainstream, to these relatively obscure OSes. This is always expensive, and takes time, and leaves you with orphan code and everything else. Anyway, to get back to your first question about my background in real time, I was a hobbyist as well. So in 1992 I brought Linux into Ready Systems, coincidentally in to the building I am in right now, an ultumate irony. I brought all the engineers into the cafeteria and said hey, I just downloaded this stuff, that's been produced on the Net by a bunch of guys, including this guy from Finland, and I built these two floppies, doing raw writes out to a Sun workstation. If you know anything about any of this, you know that the probability of anything working was zero, and stuck it into a clunker 386 that I bought at a junk show and the goddam thing worked. It came up. And it was UNIX. I was blown away. It was a very sophisticated system. We got X Windows running, put it on the network, and found acres and acres of functionality were working, very stably. Relative to the embedded software of the time, this thing was so far ahead in terms of functionality that it was an amazing eye opener. We actually did an internal study about Linux. We did a paper called VNIX - I forget the spelling, but it stood for "VRTX is not UNIX." And it was basically how to take and use Linux in an embedded operating system. We basically did a study on wha t we could do. But we had a company to run. We got busy and got merged and did a bunch of other things. But in '93 we conceived the idea of embedded Linux and how we might use it.

That would have been early.

We would have been so early. But still, it was on our minds. Every month I got my copy of Linux Journal. I was a very early subscriber.

This would have been around '94.

Yeah, one of my engineers showed me the very first edition, and I immediately subscribed. I tell the story, and it's true, that I noticed that virtually every month the magazine got thicker, the paper got better, the ads got more sophisticated, the articles got better. You could tell that something was going on. So my indirect measure of the goodness of the Linux market in general was Linux Journal. Seriously. It was my benchmark. You guys should know that you reflected what was going on.

So what happened to Ready Systems?

Well Ready Systems merged with Microtec Research in '93. Then we went public and got bought by Mentor Graphics.

And VRTX stayed with you through all this.

Yes. By the way The V and the X stood for the same VX that's in Wind River's VXWorks, which was based on the VRTX kernel. Another one of those little coincidences.

So now we're at the Mentor Graphics stage in history.

Yes. Microtec was the largest combined embedded RTOS tools company. We were larger than Wind. That was '93. In '94 we went public. Then in '95 Mentor Graphics, the EDA company, in a future-looking move, bought Microtec. And that was how the food chain worked. Then, without throwing rocks at Mentor, let's just say that the business did not prosper for any number of reasons. But I kept my subscription current in Linux Journal, and followed the Linux distributions. So at first it was Ygdrassil. No, before that, for historical accuracy, there was a distribution you could download from TSX11 at MIT and there was a brief distribution from a guy up in Canada, SLS -Soft Landing Systems. Then came Ygdrassil. They were the premier supplier for awhile, then for some reason just stopped. Then came Calderacame on the scene. And that was the first professionally packaged Linux distribution. They were on the top of the heap. Then these guys with funny hats who had crude-looking quarter page ads in the back of linux Journal came along.

Did you know that Bob Young got his Linux start here at Linux Journal?

Wow. Well, I'm not a shill for Linux Journal, the it's a big part of the story. There was an article about four or five years ago, by a guy from NASA Langley, and they had a side-looking radar system. The previous version of this thing was based on Multibus I with two CPU boards, one of which ran VRTX, my software. The other one ran some UNIX. He wrote that there was a successful system that we had managed to replace those two systems with one running Linux. And I thought, there's my stuff, good old VRTX, being replaced by Linux. Something is going on here.

You'd rather be your own cannibal.

That's right. That came completely home. VRTX was being replaced by Linux. So, between that and knowing where the customers were going, we could see Linux as a serious player in our space.

What were customers telling you?

Some of the larger VRTX customers were pushing for memory protection, because their applications were complex and needed isolation. So we started to put it together and saw that out of the box Linux wasn't going to happen. But maybe somebody with my background, plus a great engineering team, could help the market move toward what Linux does. We could close the gap. We could make Linus embeddable, by paying attention to real time, by moving Linux where it needed to be. But fundamenally, between the technical requirements and the fact that the darn thing was standard, open and full of all the good things we love about Linux, it was clear that Linux could run through the embedded market There would be tremendous opportunity.

Somehow you got out of Mentor Graphics.

We finagled to leave, yes, but we left on good terms. And immediately we went to the guys who had invested in Hunter & Ready in 1981, who by now were old friends of mine, and said "I have not come to you with a new idea in eighteen years."

"I let a generation pass."

It was true. In the RTOS space, there was nothing fundamentally new in eighteen years. Now it was time. We said we knew Linux could make it in embedded systems, and that, in all modesty, that I could make it happen. With all due respect, it was not that difficult a bet to make.That was in March of '99. For the first six months we were exactly what our competitors called us: seven guys in Sunnyvale. We didn't hide that. We were testing like crazy. Everything: customer requirements, hosts, targets, real time… everything about where we would want to go. We had some preliminary hypotheses. We tested those. And fundamentally we came down to some interesting discoveries.

Such as?

We thought that real time would be a big factor, but there were many others that were in fact more important in terms of customer acceptance that needed to be addressed first. So we re-ordered our priorities.

What were the priorities?

The fundamental one was availability. If you go to Fry's and get Linux from Red Hat or anybody else, it's a self-hosted X86 or possibley PowerPC system that works fine for installing on a PC or a Mac. But the classic paradigm for embedded systems is cross-development, where the system is hosted in one place and targetted somewhere else. An embedded processor typically is not capable of supplying its own development. So the very first thing was making Linux available and supporable as a cross-hosted environment, which means on a Solaris machine or a Linux box that would let you compile, do a sysgen, build the kernel, and also compile applications and then download them into the embedded target. This is a classic fundamental paradigm for embedded development, which is cross development. Without that you get nowhere. So the fundamental effort that Montavista put in was developing the whole architecture for how Linux exists as a cross-development system. How ou install it on a system, how the directories are laid out, how you build the compilers to be hosted on, say , Solaris but target, say, the PowerPC.

Your're talking primaily abou the develoment suite on your local host.

Right. One weird example is from a talk one of the guys gave at the Intel developer's forum a couple weeks ago. He used a target system as the system that he plugged into the projector when he gave the talk. That was a StrongARM based system, because he was talking to Intel. And his development environment for that StrongARM system was a PowerMacintosh laptop. A very odd combination, but one that falls out of our process.

The laptop was running a PPC version of Linux.

Right. It was running Yellow Dog as the host, but all the tools for cross-development for StrongARM were hosted there and ran and produced executables that ran on StrongARM. So it was a unique-in-the-universe setup.

What was the STrongARM system? Was it a board?

Actually we used an Intel evaluation board called the Assabet board. If you want to play with a StrongARM processor, you get one of these boards. But you don't run the compiler on the Assabet board. You run it on the host machine, then you dump your stuff down over Ethernet to the target board. Very typical

But there are other ways of communicating.

Oh yes. That's not the only way. You can run serial, or burn flash. All kinds of options.

With Linux you're saying you've got a Type O operating system you can run on any host, and any target. But the key is the target .And it gives you a lot of freedom as a tools developer as well.

The most natural situation is to use Linux to develop Linux. You have a simulator built in, which is the same system. But in large companies they may have 500 Sun work stations installed with Solaris. Guess what, they're not going to replace those with Linux boxes. So you need to host on Solaris. You have all the GCC tools and all the sources and everything else installed on a Solaris box, but when you punch the button and it burps out a StrongARM kerneland STrongARM executables that you can download and go. This is the way it's been done for twenty years. You cross-develop and you cross-debug. Same thing. You're going to host GDB on the host system and have the GDB server running on the target. So, to cut to the chase, we gathered all the weapons that we'd accumulated in doing 20 years of embedded systems , and re-formed Linux and its tool chain to operate in a way that embedded developers would expect. That was the first order of business. It didn't matter what you did for real time if nobody could use it. So getting out the CDK - the cross-development kit - was the first major statement of the company.

How many hosts and targets do you support?

We support four processor architectures and three different host machines.

I'm guessing the host, the development platforms,s are PowerPC,X86 and Solaris.

Yes. But there are varieties on those as well. RedHat 6.1 and 6.2 on X86. Yellow Dog on a Macintosh. And Solaris - I forget the revision number - on Sun. Then the targets are a subtle point too. There's X86/IE32 family from Intel, and as you know within that there are many variants. And the compilers are built for four or five subsets of X86 - 486, 586 and so on, that GCC - the Gnu compiler - supports. Those are all uniquely built. So, when we say we support X86, it's a broad statement. Over on PowerPC, there are even more variants, and they are more fundamentally different than the X86 variants.

I know the 601 and the604 are highly different.

Yeah, and then you get into the Motorola and IBM embedded controllers, which are great processors, that often don't have any floating point. You get into very subtle differences. In the X86 world, things tend to move back and forth relatively easily, but in the PowerPC world the differences are more distinct. These are not anybody's fault, by the way. So it took considerable effort to put together support for five or six different variants of PowerPC so that all the libraries were built correctly. So the targets without floating point didn't get floating point support. Then there were the MIPS and StrongARM processors. Together they complete the family.

I was surprised recently to discover - from your literature, in fact - that PowerPC leads in embedded shipments.

Oh yeah.

It was also interesting to see the Hitachi SuperH on the list. Many years ago I was involved in rolling out that microprocessor in the U.S. I forget the original name. One report was that it stood for Sonic the Hedgehog, from the Sega game. The SH was the embedded microprocessor in those early Sega games, and the company had a philosophy called 10, 10 10, which I think stood for 10 million, 10 MIPS and under 10 dollars, as a kind of marketing target. I may have that wrong, but the idea was that the company didn't design a major that didn't have a huge customer, like Sega.

That illustrates just one reason why there are so many different embedded processors. There are many different applications. You could kind of just cruise on the desktop doing nothing but Intel X86, but as soon as you leave that world and head into the embedded space, the first thing that hits you is that there are so many different environments to support, hosts and targets. And there is a subtlety here. If you just go out and get Linux, say, for MIPS, then for X86 and then for PowerPC, then for StrongARM, and even within those families, you have the same common source, perhaps, but the rest of it is all over the place. Linus maintains the core source tree that's X86; but after that the derivatives are extrememly varied. You can go out and support those processors and re-host them and everything else, but you'd have a Tower of Babel of different source trees for each of those different combinations. You could do okay. You could get it all out. But, what about when a bug comes along? It's in some but not in others. Now you're in a management nightmare of trying to keep everything in sync.

It would be a logical mess.

I can tell you that Microtec fell prey, even within its own product line, of having different compilers and rev levels and everything else. Even though nominally everything was supposed to be the same. So one of the core values of Monta Vista is building all of this stuff out of common source, and a common rev. We maintain a strict discipline of building all of it every night.

So you build a common source that looks across all of these host/target permutations and somehow works with all of them?

Right.

If I look at all the permutations, you've got three hosts and four targets-

-And if you include variants it's more like twelve to fifteen targets.

That comes to 45 different directions that your souce can compile.

That's correct. And if you ever get on the wrong side of that you can see how easy it is to have drifting feature sets and all kinds of conditionalities to account for.

How do you stay on the right side of that?

Software engineering.

Ha ha...

It's true. One of the things that distinguishes MontaVista, if you look at the engineering team, starting with Kevin Morgan, is that it comes out of enterprize-level, mainstream, state-of-the-art UNIX world. Kevin built very large systems, using HP-UX, with subsystems all over the world, with a very large source base to maintain. There is a true technology of software engineering that it takes to manage very large distrubted software projects like this. This is a fundamental and deep skill in the company, and it is not one that is shared with others in the Linux world. People who might be very strong in Linux – God bless them – don't necessarily have this kind of background in large-scale software engineering. That's an insight we acquired by getting it wrong, no pride here–

–That's what you have long histories for.

Absolutely. We knew full well the mistakes that could be made. The most natural way to acquire a product line as broad as ours is to have different source trees for all of it. That's the way it would come to you in the Linux world. We think – and this is no big claim, and because we're open source it doesn't matter – as far as we know we're the only place that has a uniform common source, all of the same rev level, for all the hosts and target processors we support.

Do you have a special branded name for that source tree? Is it like, your distribution?

It's standard Linux. We just did the engineering work to integrate in, and rev up, to the common rev level, all the different sub-revs of the different architectures we support. It's just engineering work to do that. The Linux community, in an asynchronous way, and in individual instances, does all this at some point. In other words, a MIPS version does get brought up to 2.1 or 2.2.something-or-other, but the StrongARM one might not be there yet. There is nothing to enforce synchrony here, nor should there be, because each interest group has its own needs and moves at its own pace. However, we are one interest group, and we want to make this synchronized release available to all our customers in a coordinated way. It's a value to the Linux community to have this synchronized rev that supports this matrix, and it happens to come from Montavista.

Are there people outside the company who have jumped in to help with the project?

Indirectly, yes. For instance on PowerPC, we have done most of the bring-ups, but not all of them. So the normal processes work there. Which means we are big contributors to the PowerPC tree and help maintain it. But we don't do it all. This is a community effort.

You believe in open source.

We are absolutely an open source advocate, so all this work is always available on the Net immediately through FTP or on CDs.

So if somebody wants to download a .tar ball of what you're doing –

–They've got it. No tricks. No hold-backs, no passwords. It's there. We have a moral obligation here. We've benefited greatly from open source and ought to give back. But beyond that we think it's a business advantage. We fully believe in it. Anything we can do to make Linux – and especially embedded Linux – go farther, faster, benefits us.

You seem to have a strong relationship at Motorola. Do you have a cooperative development relationship with them?

They have been very helpful to us as a company, making sure that reference hardware is available for us. So that as a new processor comes out, FedEx appears with a board in hand. It's gracious on their part, and we do our best to do the bringups immediately and putting them out on the Web.

There is a Motorola microprocessor in the Kerbango radio, running your Linux.

Yes, it happens to be an 823. It's a strong architecture. The IBM 405GP, for example, is also a very hot part, in the figurative sense. Both are PowerPC. We have a bunch of customers designing that part in. So we have a great relationship with IBM Microelectronics, too. We did the Linux bringup on the 405 with them. Right now we're the center of the universe for that.

Are there any embedded multiprocessing applications? I only ask because I was involved with Groupe Bull in the mid-90s, when they had finished their initial design work on the PowerPC, helping design SMP into the chip.

Yes. There are two things. Certainly in some of the server-type systems where the distinctions between embedded and standard servers are blurred, you'll see some SMP efforts. And as you know, tomorrow we're announcing a big thrust of true real-time Linux without any funny add-ons. Well, it turns out that the same work that you need to do to make Linux SMP-compliant – which is a big push for Linux because we want it to scale – is virtually identical work that you need to do to improve the preëmptability of Linux in terms of hard real-time. So rather than having to add on a different operating system, or yank the guts out of Linux to make it more real-time, we're making Linux into much more of a real-time OS.

So it can be done?

Oh yes. Because our guys have done real-time UNIXes so many times, they know exactly what to do. We jumped on the SMP work and found that we could dramatically improve the response, by a factor of thirty. That's just a huge increase in the responsiveness of Linux in the real-time domain, based on enabling, on a single processor, the SMP software, and using it to basically make Linux more responsive. But it's done with standard Linux. That's the holy grail. Just as a little editorial position, because Montavista's whole positioning is 100% pure Linux, it's not surprising that we're the guys who did this. Some of the other Linux embedded companies, such as Lynx, who have another operating system to sell you... why would they put the effort in, if it undermines their own OS.

Lineo acquired the guys who do RTAI, Zentropics, and so they think they've got a real-time solution there. Their motivation to improve Linux is very low, because they have a company trying to work a different way. Red Hat wants to sell you ECOS for real time. That's not Linux.

What's with ECOS?

That's a kind of VRTX small kernel that they have. They should have done Linux, but they didn't, and now it's too late. They've invested all their money into building ECOS and have tried to reposition it as some kind of Linux, but it isn't and everybody sees through that.

This is part of what they picked up from Cygnus, right?

Cygnus did it, and it was funded by the Japanese, who wanted a small kernel. And they did an okay job, but it's not Linux. It's a rhetorical patch. And a dead end. We're pure Linux, pure open source. We spent the money, hired the guys, and went after making Linux hard real time.

Tell me more about what it is, specifically, that you did to Linux to make it more preëmptive and SMP friendly, and somehow manage to speak to all these different instruction sets.

Good questions. Couple of things. When people think of real time, they think of "well, how fast does it respond to interrupts?" But the weakness in Linux, in terms of classic real time, is not how quickly it responds to interrupts. It actually does a pretty good job. The amount of time that the Linux kernel itself keeps interrupts turned off is reasonably small. On a 300 MHz Pentium it's probably 130 microseconds or something. It's not supersmall, but it's very very respectable. So the issue of losing interrupts is relatively trivial. The weakness in Linux actually shows up after an interrupt occurs and you want to have a process run because of that interrupt. The question is, how long would that take? The weakness is that, in standard form, it is a non-preëmptable kernel. That means that some low-priority process might have just started a system call that will take Linux a long time to finish because it does complex things. You're the pilot in the airplane and you push the eject button, and Linux gets the eject command and says "Oh, I'll queue that up, but first I'm going to go finish this low-priority thing – flushing the memory or something." Hundreds of milliseconds later, you're still waiting for action. So you have relatively long response times for completing the circuit from interrupt to actual process running.

What you want to do is make the operations in the Linux kernel itself interruptable and preëmptable. They are the former, but not the latter. The thing you want to do is stop doing the internal processing on that low-priority thing and you switch over to the process level where you get punched out of the airplane. It turns out that that process of breaking up the Linux kernel so it can leave off where it can leave where it was and come back, requires identifying sections inside the code where you can do that. This is the same thing you need to be able to do to run on multiple processors, because you want to have parallelism, where chunks of Linux run here and different chunks run there. It turns out that the same engineering investigation and thought process for SMP can be utilized to make the Linux kernel preëmptable and very much more responsive. You wouldn't know this if you hadn't done it five times over. The good news is that it's Linux. It's a side effect of the SMP work. So we're not hacking out the middle of the kernel on this. We're just enabling, on a single processor, the SMP facilities, but not for SMP work, obviously, but to enable the responsiveness. And it works like a charm.

Now that we've done this, and we've done the engineering and testing to make sure that it works, we have the same sort of capabilities as the proprietary kernels in the sense that it's preëmptable, with good interrupt off time, where it can do real time things, now it's just a battle of improving the numbers. This is like our 100 MHz version. Just like Intel, we're going to be banging on this to double the performance again, until we get to diminishing returns.

Which would be when?

When we wipe out Wind River and everybody else. (Laughing.) The important thing, however, is that we've broken through the nominal barrier that Linux had in being accepted in the embedded world, which is its hard real time capabilities as a native operating system.

What do you see as the implications of this, beyond beating competition and improving Linux?

One weird side effect shows up when you think about the desktop and playing MP3 files and doing video. Those are real-time applications. In other words, you can hear if the data doesn't get out to the speaker or the image doesn't look right on the screen. This is inherently better for Linux in general. It helps reposition the whole OS.

I suppose when your MP3 cuts out when you open email you're hearing one of those interrupts confusing the OS.

That's right. The fact is that this isn't just a better embedded Linux. It's a better Linux.

How does the rest of the community take that?

This is a good question. We're being very careful here. First, part of the charm of Linux is that you can do whatever you want with it. However, we're being very humble. This is a prototype. As we announced, our source will be on our Web site, which is our tradition. And we are offering this as a starting point for a very good set of capabilities that we are suggesting, in the most humble way, that 2.5 should consider. Linus always says "show me the code." Well, we're obeying that rule. We think this is a significant improvement in Linux in general, and showing the community the code, for all to see and play with, is the right and only thing to do.

So you're submitting it for peer review.

Absolutely. Out it goes. This is a definite improvement. In the best of the open source tradition.

Let's say that some big manufacturer wants to make smaller, cooler devices. They come to you , and you've got this free Linux.. Now... how do you make your money here?

Service. We get technically selected, just like any embedded OS company would be. Then the customers say "how do you do business?" We say, "if you want to use us, we sell, on a per-seat basis, subscriptions to Hard Hat Linux, for every engineer on the project, for one year. Like a magazine. That gets then the software they can get for free – the source, the binaries, and the rest of it, plus our expert help. This is how the RTOS business has always worked, except for where you assign the perceived value. We become your RTOS supplier, just like Wind River has always been in the past. The customer isn't interested in being in that business. They don't just want ten thousand or ten million lines of source code. They want a functional RTOS and the company behind it. So they agree to do business in the way that we want to. Any company that is serious about its work, and has time to market issues, and wants to get a product out, knows they have to do business with an RTOS supplier. Whether the code is free isn't relevant. It's naive. Our prices are industry standard. About the same as Wind or anybody else.

So they're buying a relationship.

Yes, but they were already doing that with Ready Systems and every other embedded company.

So what you're recognizing is that what you were always selling, in your earlier companies, was not the binaries, but the relationships, the expertise, the guy on the phone.

Exactly. And the road map. And everything else a customer wants to know is there. You know, when I started Hunter & Ready, every customer I talked to had their own kernel. So this is nothing new. They had the sources to their own kernels. It wasn't like there weren't any kernels around. So I would have to go into Lockheed or Northern Telecom and have them pay for something that they nominally had for free. Linux is no different.

You're recognizing that the value isn't in the code, as a product.

Right. It's make versus buy. It's better for their engineers to concentrate on their telephony applications or whatever, and better for Ready Systems, and now Montavista, to be supplying the OS. In both cases, they always had the alternative of doing the OS themselves. They just turned their back on it, because it wasn't economically feasible.

You're no cheaper now than you were then.

No. We don't charge royalties. The net/net of the business relationships between Montavista and a customer is hopefully every bit as profitable as any proprietary system – and it needs to be for us to do well.

What about royalties?

We think royalties are a very uninspired way of doing business with an engineering team. There are other ways we would rather spend more money to ensure an enduring relationship than to be so unimaginative as to count royalties. So we just don't do it. By the way, in the business nobody paid royalties anyway. They always did buy-outs or other deals that negotiated royalties away.

Explain to us how the business works, then, because we're seeing a lot of people coming into this space for the first time. I always assumed royalties were part of the business. Also customers paying for developer seats.

In the normal embedded world there is kind of a combo. Every engineer who was going to used VRTX, and the tools, would be – make up a number – $5,000 a seat. And that's normal pricing.

Ten engineers, $50,000.

Yeah. In addition, in the RTOS industry, you might try to say, "and for every copy of VRTX you use in your product, it's a buck. That would get negotiated away, one way or another.

Something you could throw in up front that would get tossed out.

It was very clear: there were buy-outs. Almost from the very first customer in 1981, they got some product, and there was a cost of goods in their product. Since they did all their own software that was all covered in engineering. The cost of software, in terms of cost of goods sold, was zero. You had an engineering cost, but in terms of incremental, per-unit cost, the software was free to them. So we walked in the door and said, "We have a great deal for you: increase the cost of your product." Well, that was crummy. Nobody wanted to do that. So we got the message pretty quickly. The customer wasn't begrudging us the money; it was just don't ask us to pay for it out of the cost of goods sold. They'd rather say "we'll pay for something out of the engineering funds." As long as they are willing to pay, why mess with royalties or any kind of per-copy thing, because you need licensing and lawyers get involved and everybody goes berserk over the numbers. We said, "Hey, we're selling to engineers. Why don't we get what we think our fair value is by having a superior product based on Linux and sold to the engineers in a way that they see the greatest value? It goes down fine. Somebody finally listened to them.

So you want to de-complicate life for the engineers.

Yeah! Another interesting side effect. If you sell these sort-of binary licensed products, you get this big pop, where they spend fifty grand with you one year to get outfitted, at the end of that year, when you want to get some more money out of them, all that's available to you is maintenance, which is twenty percent. But if you rev your product to 2.2 and force it down their throat, you can try to get another fifty grand out of them. That's the Microsoft strategy of revving Word and breaking everything with it, forcing the entire installed base to get the next version. Engineers hate that. They like stability. So the subscription model de-couples the rev of your product, which you will do from time to time in any case, from your revenue stream. How many revs happen in two years isn't the issue. Known costs and current software are the issues. Engineers want constancy and good supplier relationships. For classic embedded, with engineering involved, this is a superior way to do business.

You have fewer interrupts in the conversational bus between vendor and customer.

Absolutely. Our larger customers beat this into us. In effect, they came to us, met with us, threw our catalog in the waste basket and said "We want to do business with you, but not in that way. We want a structured, engineer-to-engineer relationship, based on your products, but much deeper than that. We want better visibility, better responsiveness, influence on your roadmap, and a closer relationship all around." We got more money from those customers rather than less.

So I would argue that you can see in the embedded Linux business the difference between the guys who have been around and the guys who haven't. The guys who have been around – that's us – approach business in this engineering way. They guys who haven't been around are trying to be Microsoft and it's just not going to happen.

Your customers then, are engineers.

We have a B2E or E2E way of working. We sell to engineering teams. That's who we automate and outfit.

I have a friend who likes to substitute the letter W for the number 2, or the preposition "to." Because they think the problem with that preposition is that it presupposes a distribution model rather than a cooperative relationship. We don't do business "to" people. We do it withthem.

That actually is a lot better. I like EWE: engineer with engineer. I could buy that. It's a multifaceted business relationship, based on technology and product – we do get technically selected – with a definite EWE quality. We don't consult. Our customers get technology seamlessly from us, in the best possible form, which is EWE. We answer the phone. They know where we're going. We know where they're going. All those things.

What this also says to me is that Linux is much more a living operating system. It's organic, and companies like yours are organs in the growing body of expertise about Linux.. With closed-source companies you have this annoying dependency, where you have to wait for the stone tablets to come down from the mountain, and you have to cope with those until the next rev. Whereas what you're doing is saying "we're suggesting this way of doing preëmtion, what do you think?"

Yes. You have to lead and participate and follow along all at once. Given the way that Linux has evolved, without central control, is so remarkable that you've got to believe that there are some forces at play that are working so well that it would be very rash to think the process could be helped along or improved. That's why we're so head-first into the process that has already produced a remarkably great operating system, and to abide by its wisdom at the same time.

This reminds me of the famous Ted Turner line, "lead, follow or get out of the way." In the Linux development community you need to be able to do all three, dynamically. You have to lead on one thing, follow on another and get out of the way on the rest.

That's true. And even in the leading part it's still appropriate to be humble and play by the rules. Once you kind of jump into this you come to understand that this is a very deep phenomenon. It has dimensions that we are probably not even aware of today.

There's not a magic formula here. We run a business.

Still, you're at not in a standard business category.

The Linux phenomenon is fundamentally different and powerful. It has cultural aspects to it, it's fascinating to be part of, and it's certainly something I'd want to bet against or be on the wrong side of, honestly.

One of the things I've observed about the Net is that it was made by this same phenomenon. It is embodied by its creators with three virtues: nobody owns it, everybody can use it, and anybody can improve it. What you're describing here is more of that same thing, and one way to make a living in it. It's in Lineo's interest to take apart your real time Linux release and put it through that same peer review process.

Yep. You're absolutely right about the Internet, and about Linux. This thing is spreading like wildfire, and its in our interest – along with everybody else's – for us to pour gasoline on it. These things have very explosive properties. And we have to contribute to the ongoing explosion. Michael Tiemann at Red Hat tells the truth when he says there is a proven model here and it's the scientific method. You do your experiments, show your results and so on.

Eric Raymnond talks about that too.

Yes. A quick anecdote. We got into a benchmark war at one of our clients. They were going to test Linux against one of the UNIX-like RTOSes. There was this caché about the non-Linux OSes that they were much higher performance in many dimensions. Well, they did an interprocess benchmark on like-to-like hardware and Linux came out five times faster. You kinda go "In a way, that's not surprising. In the whole world maybe five guys architected and engineered this closed and proprietary RTOS while we know full well the principle that Eric Raymond talks about with Linux development. A thousand eyeballs make bugs a lot more trivial. It would be surprising if Linux were not a lot faster. There were maybe a million code reviews. That's the power of this larger community. All due respect to Wind River, if you sort of squint or round off, no engineers are working on VXworks. In real number there are, but compared to Linux, the number is relatively tiny. Approximately zero.

And it's a contained number.

Right. In this unruly and wild Linux community there is something that propogates very quickly through companies like Montavista that make it possible from a business standpoint to depend upon it. It's a force. We have to be careful not to act as a filter in front of all of this, but rather to act as an enabler, so people can tap in to this phenomenon in a sane way and still get their job done. That's our purpose in life. We want to enable the engineers to do what they want to do anyway, which is use Linux. Boy, that's a deep phenomenon.

Are you saying Linux is ubiquitous with your customers?

We go into these places and virtually everybody we talk has fooled with Linux at home, at the very least. All we have to do is give them the justification to use Linux at work and you've got all these guys working on your behalf.

One of the things I observed about the IBM announcement, when they essentially declared themselves a Linux company, was that this was clearly a company that was now in full compliance with its own engineers.

Oh, there is no question that this is a phenomenon of s engineers. Look at Sun's success in the early years. There is no question that they won the hearts and minds of software engineers. If you were an engineer sitting there at a VT100 with VMS at the other end of it, and the guy down the hall had a Sun workstation with a big monitor and a mouse and multiple windows open and running UNIX, you will do everything in your power to get one of those things.

Back in the late 80s, I was involved in SPARC International, I remember talking to Brian Halla, who was with LSI Logic at the time and now runs National Semiconductor. He called Sun workstations "engineer sockets." If you hired an engineer, you had to give him a Sun workstation and a socket on the network. If you were really cool, you got to have one at home, too.

That's right. If you get on the right side of it, you see what happens, and it's just impossible to fight. Again, that's the underpinnings of what's going on. You can't argue with these guys. They knowwhat Linux can do for them. Maybe what it can't as well, but that's not what's moving this phenomenon. One of the conundrums in the RTOS industry was that it took forever to select anything because all of the systems were unknown. You'd have these long drawn-out evaluations because nobody had any clue about what any of these systems could really do. Now all of the engineers know exactly what Linux can do. Being involved in the Linux phenomenon accelerates the selection process, the training process and everything else. It cuts across so many dimensions of development. Not just technology. There's knowhow, resources, availability, emotion – people want to work with Linux. We have to harness all of that.

Was this what you told your investors?

Yes. I told them that Linux represented so much more than just a good technology, which it is. It represented a true shift in how things are going to be done, and it was time to get in on the ground floor. It was that fundamental.

I like your way of characterizing Linux as a fire you pour gasoline on. For years I liked to talk about the only kind of truly effective marketinn. The logic went like this: A) Markets are conversations; B) Conversations are fire; and therefore C) Marketing is arson. What you're saying is that if you're not committing arson, you're not really committed.

That's not a bad analogy at all. With good old VRTX, we were trying to practice marketing arson, trying to set fire to the world, and we did okay. But they were just our efforts alone. If we scaled it up a bit, we had the RTOS industry, alone, trying to set that fire. It's a respectable business. In fairness, Wind is a $300 million company through acquisition and everything. But it has taken forever to get there.

Now, however, we're part of a larger phenomenon called Linux, which is many, many conversations going on all at once. It is a wildfire. That's why it's so fundamentally different. It's instantaneously worldwide. It's very deep. It occupies the minds of countless engineers, naturally selecting the best and the brightest. It is all of these things. It's a whole lot of fun, I can tell you that.

It's a huge world where a lot of people know each other's work, or are in a position to know each other's work, and that's an advantage for everybody. Secrets can be a liability.

It is a fundamental revolution. You know, in the late Seventies Wozniac and Jobs proved that with microprocessors you could build a computer. Now Linux is proving that anybody can be an OS company. Before it was always Microsoft or Wind River or Ready Systems. Because only a few people, relatively speaking, could get the resources together to become an OS company. That's all blown out now. Anybody who grabs a Linux CD can be off to the races. We forget that when Sun got started there were a whole bunch of other 68K UNIX boxes out there. Sun's stranglehold was not that they could build a 68K box. It was that they won the loyalty of engineers. Now Linux is doing the same thing as the 68K did then. The basis of differentiation isn't that you're the only one with this OS. It's around what you do that others can't, or can't do as well.

Another metaphor worth talking about is construction. I was talking to one of your guys, Bill Weinberg, about how I thought that Linux was turning the software industry into the construction industry, not only because we use a lot of construction metaphors – tools, development, building, architecture, and so on – but because the construction industry is a good model for what the software industry becomes as it matures. There's no Microsoft there. It's a multi-trillion dollar industry where anybody can make money doing anything that makes novel or economic use of common knowledge. There are no secrets to shaping a two-by-four or building a house. And in fact new construction methods tend to spread like wildfire because the industry is one big conversation. But it isn't dominated by suppliers. It's dominated by practitioners. Just like the software industry is coming to be dominated by engineers, programmers, developers.

That's right.

Well, what surprised me was that Bill said that your version of Linux – Hard Hat Linux – was based on exactly that metaphor.

You're absolutely right. Tools and workbenches and other construction metaphors have been part of software engineering for a long time. Hey, look at Vxworks, for gosh sake. As for the name Hard Hat Linux, it's kind of a combination of that metaphor, plus the suggestion of toughness, plus a play on the Red Hat theme. It was reasonably clever and whimsical, and had a lot of possibilities for trade shows too – tool belts and whatever. By the way, we decided not to name the whole company Hard Hat software because that would have sounded too derivative of Red Hat. MontaVista is a place name. It's a part of Cupertino.

Who do you see as your competitors. Is is primarily Wind River?

In the proprietary space it's Wind. They're definitely on top. QNX to some degree. A few others. Clearly, it is tempting for customers to do their own Linuxes, to adopt a distribution. Then in the Linux space itself, the proclaimed Linux payers are Lineo, LinuxWorks and a few others. But in terms of funding and everything else, those two stand out.

LinuxWorks is an old competitor from the proprietary space. They were Lynx.

Yes. They have a product that is in the direct line of destruction by Linux. That's the Lynx RTOS. To their credit they had to do something: fold or try to bravely do something about Linux. They have been clear and explicit that they'll do something with Linux if you make them but their heart and soul is with Lynx. Believe me, it is hard enough to maintain one operating system. It's real hard with two. If you look at it, you realize a strategy like that cannot succeed. They will have a fragmented product line because they won't port Lynx everywhere Linux is. What we're doing with real time undermines that whole strategy because every incremental improvement we make to Linux diminishes the number of customers that perceive a difference between real time Linux and Lynx. The world doesn't get kinder. The world gets nastier every time we rev the product. Basically they're a company that's been around too long and finds its space being taken over by Linux. On the other hand, to their credit they are very well-funded. They will end up dividing that money in two and spending part on Lynx and part on Linux. We spend all our money on Linux.

Lineo, on the other hand, has a legacy with DR-DOS, which is also being wiped out by Linux.

Is DR-DOS still a factor?

If you look at their S-1, you'll see that a great deal of their Linux revenue is actually DR-DOS revenue. On the other hand, the founder of Lineo is the founder of Caldera, and I've always been very positive on Caldera. They were the first really nice packaged distribution. It's too bad they kind of got wiped out by Red Hat. The lesson they took away is that they will never let that happen again. So going into embedded Linux they have a shoot first, ask questions later strategy. In other words, go out, get big, and try to sort it out later. The flaw in that strategy is that you're not selling to end users. We're all in the EWE – engineer-with-Engineer – business. So you have to have substance. Buying six companies and trying to throw them all in a box doesn't guarantee substance. I'm not sure that the cut-to-the-chase strategy guarantees success in this market. In fairness, it's going to be a real battle. Technology makes satisfied engineers, and that's where we're concentrated.

I've heard you call their Linux proprietary.

It is.What they're doing is counter to the whole philosophy we've been talking about for the last thirty minutes. What makes Linux cool is not what Lineo is doing with it. They have proprietary software that's royalty-bearing, mixed into the Linux software. That indicates a sort of not-quite-getting-it situation. But they are a strong competitor, no doubt about it.

Your position, then, is pure Linux.

We're the pure Linux play. One hundred percent pure, open source and royalty-free. We have a very simple message. We absolutely live by it. We do what we say, and it's very easy for customers to understand.

Will you do acquisitions?

Sure. That's a real possibility. But I have been through two acquisitions, and I can tell you that in the best of times they are very hard to do. You get what you get, which includes good guys and bad guys, old agendas, different code bases. Lineo's acquisition of six companies has to be extremely difficult. We've grown to 100 people organically. And it shows in results. We're producing software.

Are you making money?

No. We're a start-up, and we're just trying to grow. This is a land grab. We have customers. We are doing well. But clearly when you establish European and Japanese offices, you've got a lot to eat while you ramp up. Investors don't put the money in to have it sit in the bank. They want it spent.

What does your plan show?

A turn to profitability at a future point. And we are exceeding our plan.

Can you say what you're making?

No. But I can say our revenues exceed those of some publicly traded Linux companies.

That's indicative.

You can triangulate on that.

How many customers do you have?

Fifty.

So you have EWE going with fifty customers.

They aren't contracts. They're orders. Customers issue purchase orders. They buy subscriptions. We do consult, by which I mean that customers buy NRE (non recurring engineering). But if you were to think about embedded systems, you'd think hmm... telephony equipment, internet infrastructure equipment, etc. That's who we have. Companies who make gizmos.

Do you think that the proliferation of Linux through all these formerly isolated professional specialties – consumer electronics, security systems, automotive, military, health care devices, telephony equipment – will start bringing them together a bit more? Do they start overlapping a bit? A related question: will we finally see the smart home?

We already have people look at us for the home gateway, where you have the DSL coming in to the firewall/router or whatever. You can imagine the benefit finally to all your appliances finally becoming Net-native and communicative. Maintenance, for example. Power management. There will be all sorts of unforseen cool things that you will be able to do because all the gizmos in your house will be sitting on the net where they can be diagnosed and upgraded and everything else. Let your imagination go on it. We've had discussions with people who make wash machines, both for maintenance and for repair diagnostics. The repair guy can get called out by the machine itself, then get the manual up on the LCD screen. Imagine the benefit of everything being connected, working together to save you time and money. This is easy to imagine now.

It's in Sears' interest to know our wash machine just failed.

Right. And to know what part to bring out. Or what part is failing so they can fix it before it goes. Think about power management. Everything will become more efficient. I'm a believer in things getting better in the world. And it's easy to see why.

What do you make of what we wrote about in the September issue of Linux Journal, about the intersection of embedded Linux, XML and instant messaging.

I have to say that we've been concentrating on the enabling side of making Linux work as an embedded OS. But we are in conversations now on development in that direction, so ... watch that space.

Do you think one industry will turn out to be a more significant embedded Linux category than the rest?

We see three categories. One is what we call A-to-Z embedded. That's embedded controllers and instrumentation and gadgets of all kinds. But the driver of microprocessor technology is what we call "either end of the Internet." The infrastructure equipment guys – the Ciscos and Alcatels and Nortels and fiber optic people – those are a very strong component of our business. Then the third cagegory is things like the Kerbango radio: things that attach. Appliances and all that. God bless Linux, it looks real good all over the place. You'd think, given the diversity here, that one kernel would have a hard time stretching over all these different applications and situations, but Linux is doing it. There's a communications revolution going on. The Internet is part of that and Linux is part of that, and both are driving requirements that are falling out in to general purpose embedded. It's very interesting: the two earliest customers were one infrastructure company and one appliance company: Kerbango. Both drove us to make Linux better in the same fundamental ways.

What can you tell me about the infrastructure company?

It was a telecommunications company, rack mounted, booting from flash. They needed us to get the kernel size down, booting over a net or from flash. Well, it was the same thing with Kerbango. Cross-developed in both cases. It turns out there were far more similarities than differences. The apps were different, and the hardware was wildly different, but just like both can run on a microprocessor, both can run on a form of Linux.

I see here in one of your presentations that you think the embedded Linux market will be $4.1 billion in 2010. Is that your number?

Mainly because we couldn't find one.

That's fine. You're in as good a postion to know as anybody.

True. Though we would welcome independent numbers from anybody. This number is based on software engineering population, the growth of that population, the amount of money people spend on engineers to automate design, the number of current seats. Run the numbers on that and you get something quite large. By the way, right now it's about two billion. Wind is 300 million. All the proprietary stuff adds up to about a billion. Then the customers themselves do another billion. And those are well known numbers. The idea that it can be four billion is hardly out of the question.

Is anybody calling it real time much any more.

Troublesome question. I went over to embedded because real time is one of, but not the only, characteristic of an embedded system. It's not a necessary characteristic. It's often but now always found in embedded systems. Clearly embedded and real time are not the same. But real time has a lot of caché. So we are purposely increasing our noise level on real time, because we are doing significant work here around that topic. You can also think you're smarter than your customer about what they need and end up screwing yourself, so we have to be careful.

Still, we have a new posture, which is more aggressive about real time.

Bryan Sparks, Lineo

What got you guys into this business.

I can answer two ways. The first is that we started Caldera back in '94. Back then we bought some business from Novell and started selling DR-DOS. We intended to use it as a thin client for cash registered and applications like that.

You own the original DR-DOS, then.

Caldera bought that in '96. We had a lot of interest in embedded, and DOS looked interesting to us for that. There was a need in single-board computers and non-PC-like devices.We ended up finding kind of a business there, and that's where I ended up betting my career. We spun off two companies. Caldera systems, which you know, then Lineo, then we collapsed Caldera, so there are just the two companies. It was confusing. For a while Lineo went by the awkward name of Caldera Thin Clients. We did some DOS and Linux stuff, but got more and more interest in Linux. Clearly there was something going on here. So we said "let's turn the ship here." Which we did. You won't see anything about DR-DOS on our Web site. We just don't do it anymore. Our whole business is embedded Linux.

What was the time frame here?

This was around late '98. That's when we said this is it, and picked a new name and started moving forward seriously.

Good timing.

A lot of it was luck, and being in the right place at the right time, and having revenues from the DOS business to fuel the early stages of our embedded Linux product.

You were the first out of the gate with Linux distributions too, in a way.

We were.'

And you got kinda snookered by Red Hat. I take it you're not going to let that happen again.

It's not gonna happen here. I learned a tremendous amount. It was very humbling. We were the leader. I made a lot of mistakes. We got passed in the last lap and somebody else took the checkered flag. Pick your metaphor. Boy, that's not going to happen here.

What exactly did you learn?

One was that we did not clearly have a culture that pointed at a competitor. We were leading in the space and not looking out for our back, where everybody else's guns were pointed.

You had the pioneer's problem.

There are two strategies, of course. One says that first movers always win. The other says second movers win because they take advantage of the first mover's mistakes. The second is what happened here.

Happened with DOS, too. DR-DOS was first, but MS-DOS won that race.

Here I feel like this is my second chance. And it's not just me. There are a lot of second-generation Linux people here. People from Linuxcare, even Red Hat. They've been there, done it the first time, and made mistakes we can all learn from.

And you're applying what you learned.

Absolutely. I feel that we are executing about as well as any company can. I am so proud of the people we have and the job that they're doing. Ultimately that's what wins. You can have a great pitch – our pitch is okay – but you have to execute. You need top-line growth every quarter. You need good customers who come back because you do what you say you're going to do. We have that. It's real simple stuff.

I'd like to look at the embedded space we've had for twenty or so years, and how Linux is changing that. When I first got a look at the space, around ten years ago when I was helping Hitachi launch its SH microprocessor in the States, the customer side of the business was extremely fragmented into a zillion little application areas, each with it arcane combination of hardware, application and OS. But you had to pick one and build everything on top of it vertically, in a stovepipe stack of software. There was nothing horizontal like we're trying to do now with Linux.. I'd like to hear you speak to that transition. One day the smart home is impossible, and the next it's starting to look like it might really happen.

It's amazing. We're growing and assimilating employees as fast as we can, yet the demand placed on us by semiconductor manufacturers, and their customers, still overwhelms us. So that tells me that, well, maybe Linux hasn't won, but it's certainly in extreme demand.

It's still early.

Yes, and you have to look at Wind River. If you saw their last quarter's revenue... holy flippin' mackerel. They made a ton of money. But the interest – and it's not just faddish interest – the strategic interest in Linux on embedded devices, is accelerating at Internet time. People are really grasping on to the concept. Quite honestly we struggle to keep up with demand. It's amazing. We're still a start-up, even a large start-up; but we're not large enough. There's just too much opportunity.

How many people do you have now?

Two hundred sixty. About half are here in Utah. The rest are outside.

A bunch would be through acquisitions.

We did six acquisitions this year. And we have grown staff at most of those acquisitions as well. And we don't run them as separate companies. We run them as part of our company. But the offices represented by those aquisitions have grown. We bought technology, some of which was in embryo. We had to kind of beef up that technology, but in every case the opportunity is just huge.

And you've attracted a lot of attention.

Oh yes. You'd expect me to be optimistic, but I think: gosh, if one tenth of what comes to our door pans out, we're in great shape. Linux is going to make huge inroads in this space.

Tell me more about the demand from the semiconductor guys. Does it sort out to their customers categories? Or is it something that comes out of your working relationships? Or both?

We are extremely close to our customers. Our strategy from the start has been to establish the strongest possible relationships with semiconductor manufacturers, so when we get into their roadmap, and target what would be most interesting for their customers to run Linux on. We don't want to waste time on boards that our semiconductor customers wouldn't think an attractive choice. We might solicit their help in the form of money or engineering assistance, or access to boards or whatever. We then train their sales force on what our product, in combination with their boards, allows their customers to do, that they couldn’t have done otherwise. At a hundred thousand foot view we sell a horizontal embedded product to be deployed in a whole lot of verticals.

How do you adapt to the verticals?

We verticalize some of our solutions. We’re already doing that internally. But the main idea is to provide ubiquity on embedded platforms.

Which platforms in particular?

SH 3,4, StrongARM, ARM in several variations, almost all the Motorola chips – Ncore, coldfire, Dragonball, PowerPC… Of course the X86 kind of comes for free. That comes to ten, I believe. There are others that are not as mainstream that we’re working on. Right now we’re integrating all that board support, all that work, into our tool chain. Some of those are already supported and others we have to custom-craft for each manufacturer. All of those will be targeted right out of our SDK.

So your horizontal product is Embedix.

Embedix provides a whole list of services. Not only the Linux stuff, but hard real-time. Jim Ready and others do soft real time but we do hard real time. Tymsys does too; but we do hard real time with extensions. We also do soft real time if people want that. We have a browser that we’re converting into a more generic graphical user interface with cascading style sheets, HTML, Javascript and a whole bunch of other stuff that sits on top. This all works horizontally. So as we start verticalizing some of our solutions we'll get stronger offerings in those areas. We already have design wins in routers. DSL, VPN, IP security stuff, plus verticals around some of the devices we're working on.

You say you work closely with your customers, which are primarily the semiconductor companies. Tell me more about how that works.

We assist each other in all kinds of ways. We know we're done with a project when a customer says "we've got a design win here." So we are literally part of their team. Their wins are our wins. Their customers are our customers. We want direct relaitionships with their customers, the OEM device manufacturers.

So if Motorola sells to GM or Ford, you want to be in there with them, so those companies are your customers too.

Yes. And this is the relationship we have with all the semiconductor manufacturers. They assist us, we train their people on Linux, we support the boards they want supported, we serve as a strong partner who represents them well to their customers. So they take us there, and we do.

So you want their customers deisgning with Embedix.

That's right. We do anything they want. Custom engineering, if that's appropriate. We'll acquire technology if we're a little short in some area. Or we'll build the technology if that's the better way to go. We'll hold their hand, help them do the engineering. All the professional services you need should come as part of our company.

Since you're looking out into those verticals, and out into your customers' customers, what trends to you see? Who's showing the most interest in Linux, for example?

It's across the board. But there are some surprises. Let me tell you about those. If you would have asked me a year ago, I would have told you the sweet spot for Linux was Internet appliances, routers, gateways...

Stuff close to the Net.

Right. Stuff that takes advantage of the core competencies of the Linux communities. But the areas that have been surprising for us – and we're pretty much the only ones who do this – have been in consumer electronics. I'm talking about Dragonball, MCORE ColdFire, ARM and those devices, which run in digital cameras, cell phones, handheld devices and other consumables. We are working with a lot of companies in those areas. We can really run Linux on those devices. You don't have an MMU? You constrained by 200k of RAM? We can fit in that. The interest is extremely high in that space. When you consider that our competitors in that space – CE and Palm – lead with a GUI, we're looking good with Linux, which doesn't have an inherent GUI. So we manufacture one with a browser, Java, whatever.

But the demand is for Linux in this segment.

Exactly. It starts with customers wanting to run Linux down at this tiny scale. It's fascinating.

What about process control, industrial automation and stuff like that?

It's a market, but it's not growing, and it doesn't turn over very fast. For those spaces we offer Vxworks and Psoft migration toolkits, to make the switching costs easy. Our hard real time plays real well there, too. In fact, we're doing big contracts with the government on that kind of thing. We're doing a big contract with NASA for flight simulators, weather monitoring for the FAA... things like that, through one of our acquisitions. But these aren't the surprises. The big surprises are at the bottom of the pyramid where the parts are small, the demand is high and the volumes are enormous. You see these things everywhere. So obviously those are big for us, and I think they're big for Linux too.

What's your key offering there?

An interesting road map. When you buy into Linux, you buy into a fast moving technological roadmap. We augment that with technologies we either acquire or build to make it more interesting to those verticals. It's a fascinating thing. Previously they were either buying from Palm or CE, which are proprietary RTOSes with no road map at all.

What is your position among your competitors, in respect to customers.

I can't imagine anybody talking to more customers and partners than we are, just because we're large. We get a lot of feedback on what customers are asking for. Right now we're busy just backfilling customer requests on a road map. We have to get ahead of that. We need to lead in embedded Linux, from the customer's perspective. Right now we're just satisfying customer requests. That kind of drives our road map. But eventually we'll have to do more R as well as D. I think we'll get to the R in six months or so when we get ahead of the curve a bit.

I would imagine that the D kind of pulls the R along.

The customers are telling us "we need this," and we have a choice to make. Is it on our road map? No? What do we do? Build? Buy? License?

What's an example of that?

Bluetooth – the hundred-meter wireless Ethernet speed system. The question was, do we build a bluetooth stack? Because we were thinking about building one. Them out of nowhere IBM announced an open source Bluetooth stack. Thank you very much.

How do you contrast what you're doing with what Montavista is doing?

First, we're not a pure open source play. We license third party products with IP – intellectual property. We have our own IP. We don't open source our browser. It's really good, and we don't see any reason to open source it. The benefits don't outweigh the liabilities. I guess the main differentiation is that we lead with product and backfill with service. We are highly motivated to build a Wind River-like tool chain. An SDK very comparable to Tornado, but embedding Linux as a target. Giving the customer more and more and more tools. On the platform that they want – NT, Linux, whatever, but targetting Linux devices. We then backfill with professional services. Custom engineering, whatever. And from what I can tell, Montavista leads with service, custom engineering and soforth. Because if you're a pure play with open source, what else do you do?

What you're about, really, is tools.

You will see an ever-increasing series of tool chain enhancements coming out of Lineo. A bigger and bigger SDK.

So your orientation to open source is to cherry-pick those products and tools that are useful to you and building a suite of products, primarily tools, for shaping Linux into something useful for your customers.

Those tools we build. But what I'm talking about are design automation tools. Graphic configurators. Things that run on the device. Embedded apps. Our question constantly is, "How can we make the time-to market for our customers as short as possible?" To the degree that we're successful there, we will steal business away from Wind River. We will become an attractive alternative. The fact that we run Linux gets us in a lot of doors, because everybody is looking at Linux for embedded devices. But once we're in there, we need a full breadth of offerings. That's why we spend a lot of money on acquisitions, to make sure we cover that breadth. If you are embedding Linux on a device, no matter what platform it's on, or what the device is, we're the people to talk to. We cover from very very small all the way up to high end availability. Want Linux on a device? Any device? We're it.

The center of the universe for us is the customer. The engineer at an OEM. That engineer needs a very high quality set of tools. Compilers. Graphical configurators. Graphical debuggers, remote debuggers, heuristic tools, design automation tools, whatever. And we're going to win.

It sounds to me like you're building a Cisco.

We're not doing hardware.

I was thinking of the kinds of customers you're working with, your approach to acquisitions, your determination to lead the market no matter what, your customer focus. I'm amazed, for example, at Cisco's ability to acquire companies and not have trouble swallowing them.

Right. You can imagine that we had a strategy for acquiring companies while we were private. We wanted to learn how to integrate. We started our Cisco integration game when there weren't that many people looking in our underware.

Because when you're public everybody is looking. And you know what? Buying companies is hard. Signing the document is easy. Integrating personalities, cultures, ownership responsibilities, technologies, all those hard grungy things. We learned a lot. We didn't lose anybody in the process, but it was a lot harder than I thought. It took us two months more than I thought it would take.

That's still fast. Lineo is a young company. Haven't you acquired all those companies just this year?

All this year, within a four-month period. You said it right. I had a strong notion of the culture I wanted in our company, but we weren't of a size made it readily apparent that when we bought a company that our culture dominated. We acquired six companies that in aggregate were as big as we were. So here we are trying to just get on the same page for mission, core values, goals, cultural stuff. Now I think we have a well-defined culture, a good start... Now we're ready for the next one. We know what questions to ask, how to integrate. We are putting together an integration team, so we're truly ready and acquisition is a core competency for us. It's not cheap to build or buy.

The percentage of acquisitions that fail, in general, is pretty high. What makes Cisco stand out for me is that they manage to do it pretty well. If you guys have no major hiccups, maybe something is going right there.

It's a tough thing. We're not just a roll-up. We do want to know how to acquire, however. And we're learning. I have grand aspirations for Lineo, and I think we have the management team, and the board, to allow us to go after that grand vision. We don't want to be just another Linux player. We want to lead in the embedded software market, with Linux as our shield, leading the charge.

It seems to me you went about your acquisition strategy based on target device size scale, from small to large.

That's exactly right. We said "We want to do that. How do we get there?" We had holes, just working with a generic embedded OS. We didn't have anybody at Lineo with real time experience. We needed to go acquire that expertise.

How do you integrate Linux and your hard real-time capability. Do you have a name for that?

It's folded into our Embedix product, and it's a feature of our tool chain. You want real time on this device? Here's how you select and configure it.

So when an engineer goes about designing for a particular processor, application and board, and real time is one of the things he wants, he implements it through Embedix tools. But it's not Linux, necessarily.

Yes. But that, by the way, we open source. That whole thing is available to everybody. We followed the RTAI – real time application interface – standard. We open sourced all of that. You'll be seeing more from us there. Actually we were ahead in real time for a while, but let others move in and define the market.

Who?

Jim (Ready). He talks about it all the time. And he doesn't even do hard real time.

They announced their preëmptive kernel yesterday, which they're vetting with the Linux development commmunity.

Yeah, he's kind of defining that space, but we're going to fix that. That's why we acquired Xentropics, because they're the ones who did our hard real time.

Where do you see Linux moving the embedded space?

I think by offering Linux on embedded devices we raise the bar on functionality that customers will now require. There is this dichotomy in the world where a lot of companies say they are customer driven, but in fact customers only ask for what they are aware of. They don't know their necessities until they become aware of what's really available. The context changes. Adding Linux to Internet devices of all types will seriously raise the bar in functionality that people will demand. An example. In our first release of Embedix early this year, we matched Vxworks in functionality. In our fist release. I am on the boat with the Linux community and the network effects that brings. Look at the feature/function curve relative to time. Ours is much steeper than Wind River's. It's real simple. I don't have to incur all the development costs to get a feature, because I leverage the network effect of Linux. It gives me a nice base so I can concentrate on the really interesting embedded augmentations. We're gonna blow them away. They will have to spend more development dollars to match our functionality, while we're busy moving the whole space at a much lower cost. We're raising the bar constantly.

I'd like to go back to the power of invention. The real marketing principle isn't "necessity is the mother of invention," but "invention is the mother of necessity."

That's right. We want to show customers what they're missing out on. Linux does that on a huge scale.

It also seems to me that Linux has the same network effect that the Net has had, only starting a bit later. It's kind of the other shoe dropping. Same phenomenon, just a later stage.

Our mission statement is, "we unleash the imagination of our customers to innovate for their customers." That means we want our customers to innnovate in ways they didn't even think they could before.

So your inventions lower the thresholds of their inventions.

Yes. There's another thing. We are going to provide a flexibility that these guys have never seen before. Before Linux came along, and a customer needed a feature, they went to Wind River and eighteen months later they might get it. The whole thing is way too stymied. Between Linux and our SDK we're really going to unleash the imagination of our customers. And they will only demand more and more features, which we will be in the best position to provide. We want their time to market requirements set the pace. Not ours.

You mentioned doing this surprising work with small consumable devices. Are you working with some of the Asian consumer electronics guys?

Nearly all of them. We have fifty people in Asia. A large presence in Japan, plus another fifteen or so in Taiwan. Most of the investments we did in our round of financing came from Asain manufacturers. Acer, Samsung, Mitsubishi, a bunch of others. And we are working with a bunch of people. I go to Asia a lot. Fortunately we're kind of all alone over there. It's hard to get in. We got in partly by acquiring a company in Japan that was doing embedded custom engineering. Then we organically grew a Taiwanese office with native personnel.

I'm interested in scale. A huge company like Hitachi might have a huge relationship with a customer like Sega. How is there room for guys like you in there? I guess there was room for Wind River, though.

I don't think our business model is all that different from Wind River's. They, being such a dominant supplier with few competitors in the space, were able to extract more money from the semiconductor manufacturers than we can; but that's part of our strategy – to commoditize the space a little bit. When they talk to investors, they talk about raising per-unit value. When I talk to investors, I say we're here to commoditize system software for interenet devices. The model is the same, but we're really putting price pressure on them. We not only give them real competition, but make it much harder by coming in at a lower price.

You still have enough room in there to make money on tools and services.

We do request a small royalty, adding our own third party intellectural property on the device. But again our strategy is customer acquisition through commoditization of the space, sell tools, and grow. But in terms of what we sell, we're similar to Wind River. It's just that a smaller percentage comes from royalties.

You charge on a per-workstation basis for the SDK.

Yes, and we have royalty-bearing components in there.

You pass those costs through.

That's right. And that's very standard for the industry. We may do more licensing. We may do more acquisitions in the area so we can keep the price low but still provide the functionality so we don't have to carry the royalties back.

How big do you expect this category to get?

There are a bunch of analysts who say big, big, big. We want to grow faster than the market as well. Just keeping pace with the growth of the pie, we're at a rate of 60%

Do you see The Smart Home finally coming, because of Linux's standardization properties? Or will the firewalls of ignorance that surrounds every specialty going to still prevent that from happening?

I don't know. I brought a bunch of CAT-5 cable to the drop point at my house and the phone guy made such a hairball of it that I had to redo it myself. But some of the standards are helping a bit. HPNA, a home networking protocol that runs over twisted pair, helps. Bluetooth helps. We're working with a company we're hoping to close that is doing stuff with home gateways like I never imagined. We have a TV set top solution that we've sold quite a bit. But this is something else. We're talking about something wired to appliances that does far more than turn them on and off. It does service and warranty control. Your AC unit can say its compressor is going and to send for a guy to avoid the insurance hit. That kind of stuff. I'm crossing my fingers that we get that one closed because it's just so cool.

That's part of the promise of XML. Devices can talk to devices, live, in an XML stream.

Can you imagine this little control box calling a technician without the customer knowing what's happening until the technician says "I'm here to fix your fridge." This isn't ten years out. It's next year. We giggled a couple years ago that somebody in Japan put a browser on a fridge. Now, so what? We can put one on the microwave. There's your recipe. Your documentation. There was a piece out of Xerox PARC a long time ago that was right on. We are living that now. And Linux will only help there. Great internet infrastructure. Remote diagnostics. Network management. VPNs built in. If you're a telecommuter, Linux is great. We actually sell a little hardware box that's a VPN router. We sell it like candy. It's unbelievable.

You sell a lot of set top boxes?

Yeah. You can browse the Web on these things, watch enhanced TV, get your email, or whatever. We sell a boatload of that.

Are you using Mozilla on your set top box, like Intel is doing?

We've packaged a Mozilla with some embedded Linux, but it requires some kind of hardware assist so you can see it on a TV.

But your browser is different, from the ground up?

It is. We started building one almost three years ago, because there wasn't one for TV output. Then Spyglass came out with one. Now we offer the whole solution. And you can use whatever capabilities you want. Maybe you only need HTML, not HTTP. Maybe you just need the Javascript. We're componentizing our browser for our customers so they can pick and choose. We want to do other things, like XML. That's one reason Jabber's kind of interesting. We already have a technology we can integrate that with. All that stuff is really cool.

What's the hardest part of your job right now?

Prioritizing. Part of it is intuition. Opportunity. Unfortunately, we have to say no to some people. We need to be able to execute.

What else do you open source besides your real time component?

Go to www.opensource.lineo.com. There's quite a bit there. Utilities that run on an embedded device, like BusyBox. A bunch of drivers that we think would be interesting to others. We have a pile of open source projects.

Are your fingerprints going back on the Linux kernel?

Oh yeah. We give a lot of patches back. We want them to take more. But sometimes the patches we have, to fit on very small devices, aren't very mainstream.

What has Linus done for the embedded Linux space, by talking up mobile and other forms of development, and working with Transmeta?

I think he's done a tremendous amount. He's helped Transmeta a great deal too, giving them branding and recognition. And he really seems to enjoy it there.

Last question: What kinds of companies will survive and what kinds will not?

I think there will be tremendous revenue opportunities in this space. I cannot imagine that we'll be the only ones succeeding here. Lots of new companies seem to be rushing in, but it will be tough for a lot of those. We think we have found an interesting model that investors and investment bankers like, and we'll see if we can keep executing on it. But we won't be alone. And don't count out the larger companies – Red Hat and others – will make some moves.

Red Hat has the Cygnus thing.

Well, you'd think. What they're doing, however, is classic defensive stuff. They do everything. But I don't see them focussing on any one thing. You know, Yahoo got really big, but that didn't stop eBay from focussing on just one area. I wish Red Hat success at what they're doing, but we're going to make a living in this embedded space. This is all we do. The more it looks like a PC, the less we're interested.

Any last thing you'd want to add?

You'll like this one. The cluetrain stopped at Lineo, and by God, we got on.